Diagnosis of a disease condition using an automated diagnostic model

ABSTRACT

A method of identifying an object of interest can comprise obtaining first samples of an intensity distribution of one or more object of interest, obtaining second samples of an intensity distribution of confounder objects, transforming the first and second samples into an appropriate first space, performing dimension reduction on the transformed first and second samples, whereby the dimension reduction of the transformed first and second samples generates an object detector, transforming one or more of the digital images into the first space, performing dimension reduction on the transformed digital images, whereby the dimension reduction of the transformed digital images generates one or more reduced images, classifying one or more pixels of the one or more reduced images based on a comparison with the object detector, and identifying one or more objects of interest from the classified pixels.

CROSS REFERENCE TO RELATED PATENT APPLICATION

This application is a continuation of U.S. patent application Ser. No.13/992,552, filed Jul. 25, 2013, which is a U.S. National PhaseApplication of International Application No. PCT/US2011/063537, filedDec. 6, 2011, which claims priority to U.S. Provisional PatentApplication No. 61/420,497, filed Dec. 7, 2010, all of which areincorporated herein fully by this reference.

BACKGROUND

Automated detection of objects of interest in images has rapidlyexpanded in the last decade. For example, lesions caused by disorders ofthe eye, especially Diabetic Retinopathy (DR) and Age-related MacularDegeneration (AMD) can be detected through image analysis. One importantmotivation is the need for reproducible, efficient, early detection, orscreening, of large at-risk populations. In particular, there is apreponderance of evidence that early detection and timely treatment ofDR, the most common cause of blindness in the working age population ofthe United States and of the European Union, can prevent visual loss andblindness in patients with diabetes.

Most conventional efforts have focused on automatically detecting thelesions in color fundus photographs, as this is a non-invasive andcost-effective modality. One study of a large screening dataset hasshown that, even though some automated detection algorithms achieve adetection performance comparable to that of human experts, furtherimprovements are desirable before they translation into clinicalpractice. A primary step in automated detection of DR is the detectionof microaneurysms, which are highly pathognomic and often the first signof DR, although it has been shown that detecting other lesions canimprove DR detection performance. However, current methods fail tosufficiently differentiate lesions from retinal blood vessels: when amore sensitive setting was used, false positives occurred on thevessels, while when a specific setting was used, lesions connected to orclose to the vasculature were missed.

Several automated detection algorithms for detecting the lesions of AMD,the most common macular disease associated with aging and the mostimportant cause of blindness and visual loss in the developed world,have also been published. In particular, these efforts have focused ondrusen, the hallmark of AMD. One of the main challenges in detectingdrusen is the ability to differentiate them from bright lesions causedby other diseases, in particular, DR's exudates and cotton wool spots,as well as subtle, drusen-like lesions such as the flecks related toStargardt's disease. AMD tends to occur in older patients thanStargardt's disease, but both the individual drusen and flecks, as wellas their distribution, are very similar.

Negative lesion confounders are typically structures that are similarlooking, but not the target lesion such as vessel portions in themicroaneurysm case and exudates, cotton-wool spots and Stargardt flecksin the drusen case. Positive lesion confounders are typically a subsetof target lesions that are easily missed because they have specificproperties that are substantially different from the standard case, butshould still be detected. In the case of microaneurysm detection, theseare microaneurysms connected to the vasculature, for example. In otherwords, negative lesion confounders are the false positives of simpledetectors and positive lesion confounders are the false negatives ofsimple detectors. Typically, lesions are hardly detected if they are notspecifically modeled. In both cases, lesions that are proximate to otherlesions, including being fused with these other lesions, also arepositive lesion confounders.

Additionally, retinal vessel bifurcations are important landmarks inretinal images.

Their metrics are important for clinical evaluation as well as input fordownstream image processing, such as retinal image registration andbranching analysis. Because of their geometrical stability, they couldbe the feature points for image registration and change detection. Theyare important targets in the vessel segmentation problem and analysis ofthe retinal vessel trees. Accordingly, system and methods for detectingand/or differentiating objects of interest such as lesions,bifurcations, abnormalities, and/or other identifiable objects orcollections of pixels, are desirable.

SUMMARY

It is to be understood that both the following general description andthe following detailed description are exemplary and explanatory onlyand are not restrictive, as claimed. Provided are methods and systemsfor optimal, user-friendly, object background separation have beendeveloped, wherein the system and method provide a means toindicate/detect characteristics of an image such as legions evidencing adisease.

The present invention is a new, fast way to detect objects in images,where the detectors for these objects can be optimized. It can be usedin a variety of applications, such as to detect abnormalities and otherstructures in retinal images, in other medical images, and in any otherimages, including robotic and computer vision as well as in onlinesearch. For example, very good results are achieved in images of theretina, of blood cells, of fruit trees, and in satellite images ofhelicopter landing pads.

For example if one needs to find apples in images taken from theInternet, the user would indicate a few example apples in a few images,the algorithm then finds the optimal detectors for such apples, and canbe used to quickly find apples in millions of images. Another advantagetherefore is that no expert knowledge of the algorithm or imageprocessing is needed, and the user only needs to be able to indicate theobjects that are to be found.

With respect to retinal imaging, one problem with prior approaches isthat normal tissue which resembles abnormalities were detected asabnormalities, while other abnormalities lying close to specificstructures such as vessels are missed. In other words, the detectors arenot optimal. The new method of the present invention is substantiallyfaster, and finally mimics quite closely the processing of visualinformation by the brain. One reason for these improvements is the useof optimal detectors, and these are developed in the new approach in avery systematic way.

In an aspect, an optimal filter framework is provided to solve lesionsegmentation and differentiation problems in retinal images, includingbut not limited to microaneurysm detection and drusen versus flecksdifferentiation. As an example, the optimal filter framework can providedetection that is as close to instantaneous as possible. As a furtherexample, the optimal filter framework can allow instantaneous feedbackto the camera operator, as well as immediate feedback of the detectionresults to the patient.

In an aspect, the optimal filter framework can automatically generatefeatures adapted for differentiating target lesions, including atypicaltarget lesions, from negative lesion confounders. As an example, theoptimal filter framework can rely on the generation of a set of filtersmatched both to the lesions and to common positive as well as negativelesion confounders.

In an aspect, the design of an exemplary filter can be defined as one ofthe following:

-   -   1) expert-driven: a mathematical model is designed by a human        modeler for both lesions and lesion confounders, and the model        parameter space is subsequently traversed to generate        (different) artificial image samples    -   2) data-driven: a set of annotated image samples is used.

In an aspect, modeling the lesions mathematically is convenient in thesense that no training images are required: the human brain is able togeneralize out verbal descriptions by clinicians, though a limitednumber of samples can serve the modeler (usually an experiencedprogrammer) well. Moreover, no additional annotations are required eachtime the method is applied to a new dataset. On the other hand, decidinghow to model the target lesions is subjective and biased, and a gooddescription may not (yet) be available. Therefore, fully automating thedesign is attractive, as it makes the proposed approach immediatelyapplicable to a wider range of lesion detection problems, and anymodeling bias is avoided. The purpose of this paper is to illustrate theproposed optimal filter framework and perform numerical tests ofdetection performance of detecting microaneurysms and drusen in retinalimages.

In an aspect, a method of identifying an object of interest in digitalimages can comprise: obtaining first samples of an intensitydistribution of one or more objects of interest in one or more of thedigital images based upon one or more wavelength bands; obtaining secondsamples of an intensity distribution of confounder objects in one ormore of the digital images, at a predetermined frequency; transformingthe first and second samples into an appropriate first space; performingdimension reduction on the transformed first and second samples, wherebythe dimension reduction of the transformed first and second samplesgenerates an object detector; transforming one or more of the digitalimages into the first space; performing dimension reduction on thetransformed digital images, whereby the dimension reduction of thetransformed digital images generates one or more reduced images;classifying one or more pixels of the one or more reduced images basedon a comparison with the object detector; and identifying one or moreobjects of interest in the reduced digital images from the classifiedpixels.

In another aspect, a method of identifying an object of interest in aplurality of digital images configured as an image group can comprise:obtaining first samples of an intensity distribution of one or moreobjects of interest in the image group based upon one or more wavelengthbands; obtaining second samples of the intensity distribution ofconfounder objects in the image group, at a frequency high enough toaffect a pre-defined performance metric; transforming the first andsecond samples into an appropriate first space; performing dimensionreduction on the transformed first and second samples, whereby thedimension reduction of the transformed first and second samplesgenerates an object detector; transforming the image group into thefirst space; projecting the transformed image group into the reduceddimension space to generate a reduced image group; classifying eachpixel in the reduced image group with an appropriate neighborhood basedon a comparison with the object detector; and automatically identifyingone or more objects of interest in the reduced image group from abnormalpixels based upon the comparison with the object detector.

In a further aspect, a system can comprise: a memory for storing digitalimages; and a processor in communication with the memory, the processorconfigured to: obtain first samples of an intensity distribution of oneor more objects of interest in one or more of the digital images basedupon one or more wavelength bands; obtain second samples of an intensitydistribution of confounder objects in one or more of the digital images,at a predetermined frequency; transform the first and second samplesinto an appropriate first space; perform dimension reduction on thetransformed first and second samples, whereby the dimension reduction ofthe transformed first and second samples generates an object detector;transform one or more of the digital images into the first space;perform dimension reduction on the transformed digital images, wherebythe dimension reduction of the transformed digital images generates oneor more reduced images; classify one or more pixels of the one or morereduced images based on a comparison with the object detector; andidentify one or more objects of interest in the reduced digital imagesfrom the classified pixels.

Additional advantages will be set forth in part in the description whichfollows or may be learned by practice. The advantages will be realizedand attained by means of the elements and combinations particularlypointed out in the appended claims. It is to be understood that both theforegoing general description and the following detailed description areexemplary and explanatory only and are not restrictive, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate embodiments and together with thedescription, serve to explain the principles of the methods and systems:

FIG. 1 is a flow diagram of an exemplary method;

FIG. 2 is an exemplary representation of image samples;

FIG. 3 is a flow diagram of an exemplary method;

FIG. 4 is a flow diagram of an exemplary method;

FIG. 5 is a flow diagram of an exemplary method;

FIG. 6 is a block diagram of an exemplary computing system;

FIGS. 7A-7D are exemplary image samples;

FIG. 8 is a representation of an exemplary image determination;

FIG. 9 is a graph of FROC of microaneurysm detection;

FIGS. 10A-10D are exemplary graphs of time complexity assessments;

FIG. 11 is an exemplary graph of ROC of microaneurysm detection; and

FIG. 12 is an exemplary graph of ROC of drusen versus flecksclassification.

DETAILED DESCRIPTION

Before the present methods and systems are disclosed and described, itis to be understood that the methods and systems are not limited tospecific synthetic methods, specific components, or to particularcompositions. It is also to be understood that the terminology usedherein is for the purpose of describing particular embodiments only andis not intended to be limiting.

As used in the specification and the appended claims, the singular forms“a,” “an” and “the” include plural referents unless the context clearlydictates otherwise. Ranges may be expressed herein as from “about” oneparticular value, and/or to “about” another particular value. When sucha range is expressed, another embodiment includes from the oneparticular value and/or to the other particular value. Similarly, whenvalues are expressed as approximations, by use of the antecedent“about,” it will be understood that the particular value forms anotherembodiment. It will be further understood that the endpoints of each ofthe ranges are significant both in relation to the other endpoint, andindependently of the other endpoint.

“Optional” or “optionally” means that the subsequently described eventor circumstance may or may not occur, and that the description includesinstances where said event or circumstance occurs and instances where itdoes not.

‘Optimal’ means that objects in background are detected equal than orbetter than any other method using the same user input.

An “object” is defined as: a regular or irregular region in an imagewith defined borders contained within a digital image.

An “object of interest” is defined as an object that most closelyresembles the concept that the user(s) are interested in detecting inbackground. In the case of medical diagnosis, these may be lesions orabnormalities. In the case of retinal diagnosis, these may include butnot limited to: microaneurysms, dot hemorrhages, flame-shapedhemorrhages, sub-intimal hemorrhages, sub-retinal hemorrhages,pre-retinal hemorrhages, micro-infarctions, cotton-wool spots, andyellow exudates.

A “feature” is defined as: a group of one or more descriptivecharacteristics of objects of interest

A “set of features” is defined as: a customized group of one or moredescriptive characteristics of objects of interest

A “threshold” is defined as: a level, point, or value above whichsomething is true or will take place and below which it is not or willnot, such levels, points, or values include probabilities, sizes inpixels, and values representing pixel brightness, or similarity betweenfeatures.

“Background” is defined as all the regions in all images that are notobjects of interest.

A “supervised procedure” is defined as: a computer programming methodwherein a program learns a function from examples of inputs and outputs.

Throughout the description and claims of this specification, the word“comprise” and variations of the word, such as “comprising” and“comprises,” means “including but not limited to,” and is not intendedto exclude, for example, other additives, components, integers or steps.“Exemplary” means “an example of” and is not intended to convey anindication of a preferred or ideal embodiment. “Such as” is not used ina restrictive sense, but for explanatory purposes.

Disclosed are components that can be used to perform the disclosedmethods and systems. These and other components are disclosed herein,and it is understood that when combinations, subsets, interactions,groups, etc. of these components are disclosed that while specificreference of each various individual and collective combinations andpermutation of these may not be explicitly disclosed, each isspecifically contemplated and described herein, for all methods andsystems. This applies to all aspects of this application including, butnot limited to, steps in disclosed methods. Thus, if there are a varietyof additional steps that can be performed it is understood that each ofthese additional steps can be performed with any specific embodiment orcombination of embodiments of the disclosed methods.

The present methods and systems may be understood more readily byreference to the following detailed description of preferred embodimentsand the Examples included therein and to the Figures and their previousand following description.

As will be appreciated by one skilled in the art, the methods andsystems may take the form of an entirely hardware embodiment, anentirely software embodiment, or an embodiment combining software andhardware aspects. Furthermore, the methods and systems may take the formof a computer program product on a computer-readable storage mediumhaving computer-readable program instructions (e.g., computer software)embodied in the storage medium. More particularly, the present methodsand systems may take the form of web-implemented computer software. Anysuitable computer-readable storage medium may be utilized including harddisks, CD-ROMs, optical storage devices, or magnetic storage devices.

Embodiments of the methods and systems are described below withreference to block diagrams and flowchart illustrations of methods,systems, apparatuses and computer program products. It will beunderstood that each block of the block diagrams and flowchartillustrations, and combinations of blocks in the block diagrams andflowchart illustrations, respectively, can be implemented by computerprogram instructions. These computer program instructions may be loadedonto a general purpose computer, special purpose computer, or otherprogrammable data processing apparatus to produce a machine, such thatthe instructions which execute on the computer or other programmabledata processing apparatus create a means for implementing the functionsspecified in the flowchart block or blocks.

These computer program instructions may also be stored in acomputer-readable memory that can direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablememory produce an article of manufacture including computer-readableinstructions for implementing the function specified in the flowchartblock or blocks. The computer program instructions may also be loadedonto a computer or other programmable data processing apparatus to causea series of operational steps to be performed on the computer or otherprogrammable apparatus to produce a computer-implemented process suchthat the instructions that execute on the computer or other programmableapparatus provide steps for implementing the functions specified in theflowchart block or blocks.

Accordingly, blocks of the block diagrams and flowchart illustrationssupport combinations of means for performing the specified functions,combinations of steps for performing the specified functions and programinstruction means for performing the specified functions. It will alsobe understood that each block of the block diagrams and flowchartillustrations, and combinations of blocks in the block diagrams andflowchart illustrations, can be implemented by special purposehardware-based computer systems that perform the specified functions orsteps, or combinations of special purpose hardware and computerinstructions.

FIG. 1 is a block diagram illustrating an exemplary method. Thisexemplary method is only an example of an operative method and is notintended to suggest any limitation as to the scope of use orfunctionality of operating environment architecture. Neither should theoperating environment be interpreted as having any dependency orrequirement relating to any one or combination of components illustratedin the exemplary operating environment.

In an aspect, the method 100, illustrated in FIG. 1, can comprisegenerating an optimal set of filters, at 101, to differentiate objectsof interest such as lesions and positive and negative lesionconfounders, for example. As an example, an optimal filter framework fordetecting target lesions can comprise obtaining a representative set ofsamples from one or more of lesions, positive lesion confounders, andnegative lesion confounders, at 102. As an example, a set of filters, ora basis, can be derived from the samples to optimally represent thesamples. Although the description of FIG. 1 is directed to lesions andlesion confounders, other objects of interests can be detected anddifferentiated in a similar manner using the systems and methodsdescribed herein.

In an aspect, the representative set of samples can uniformly describethe space of all possible samples (e.g., lesion samples), as well as thespace of possible confounders (e.g., negative lesion). As an example,the most common lesion confounders can be sampled.

In an aspect, the design of an exemplary filter can be defined as one ofthe following:

-   -   1) expert-driven: a mathematical model can be designed by a        modeler for both lesions and lesion confounders, and the model        parameter space can be subsequently traversed to generate        (different) artificial image samples; and    -   2) data-driven: a set of annotated image samples can be used.

In an aspect, the method of FIG. 1 can comprise a training stage and aclassification or test stage as shown, as shown in FIG. 1. As anexample, during the training stage, a set of images (e.g., digitalimages with expert-annotated objects), can be used to build one or moreobject filters, and to optimize the classification.

In an aspect, internally the training set of images can be divided intoa first subset and a second subset. As an example, the first subset canbe used to build an optimal set of object filters. As a further example,the second subset can be used to perform feature selection on theoptimal set of object filter, resulting in selected filters. Aclassifier can be trained based on the selected filters, resulting in atrained vision machine for which the performance can be determined.

In an aspect, during a test stage, the objects in a test images set(e.g., retinal images) are detected and localized by the system and arecompared to the annotation of an expert. As an example, to maintaingenerality and robustness, one or more of the first and second subsetsand the filters are representative samples from the real data that arekept entirely separated.

In an aspect, by the proposed framework(s) described herein, a set offilters can be obtained to form a feature space and reference points inthe feature space. As an example, an unknown fundus image underdetection can be first transferred into the feature space and each pixelcan be assigned the probability of being a bifurcation, thus a softprobability map of bifurcations can be generated.

The following description details the exemplary steps of various aspectsof the proposed framework(s), including the sample generation, filtergeneration, bifurcation detection and feature selection.

As explained above, both/either an expert driven (e.g., mathematicalmodeling 104) and/or data-driven design (e.g., direct sampling 106)approach can be used in order to obtain samples representative of theobjects (e.g., lesion) of interest, as well as samples from positivelesions confounders, negative lesions confounders or any combination ofthese.

As an example, in the expert-driven approach (e.g., mathematicalmodeling 104), a modeler can be configured to translate verbaldescriptions with clinicians, as well as intensity distributions in alimited number of samples selected by the clinicians, into mathematicalequations. In an aspect, the mathematical equations can be based upon acontinuous model of the intensity distribution of the modeled lesions,and were defined for typical lesions, for positive lesion confounders ifany are identified, and for negative lesion confounders, if any areidentified. As a further example, the intensity profile of the lesionscan be modeled by generalized Gaussian functions of normalizedintensity, for example, such as the following function:profile(r;β,δ)=δe ^(−r) ^(β)

where r is the distance to the center of the profile divided by itsstandard deviation, β characterizes the sharpness of the contours and δis the amplitude (δ=1 for drusen, δ=−1 for microaneurysms).

In an aspect, the typical intensity distribution for both microaneurysmsand drusen can be modeled as:

$\begin{matrix}{{{typical}\left( {x,{y;\alpha_{1}},\alpha_{2},\beta,\theta} \right)} = {{profile}\left( {{r;\beta},\delta} \right)}} \\{u = {{x\;{\cos(\theta)}} + {y\;{\sin(\theta)}}}} \\{v = {{{- x}\;{\sin(\theta)}} + {y\;{\cos(\theta)}}}} \\{r = \sqrt{\left( \frac{u}{\alpha_{1}} \right)^{2} + \left( \frac{v}{\alpha_{2}} \right)^{2}}}\end{matrix}$

As an illustrative example, a plurality of image samples and samplesderived from the models are provided in FIG. 2. Other images in variousdomains can be processed in a similar manner. Examples of image samples(first row 202) and similar samples derived from mathematical models inthe spatial domain (second row 204) and after convolution with Haarfilters (third row 206: vertical Haar filter, fourth row 208: horizontalHaar filter). The first six samples can come from the microaneurysmdetection problem: a typical lesion (a), positive lesion confounders(b), (c) and negative lesion confounders (d), (e), (f). The last twosamples come from the drusen (g) versus flecks (h) classificationproblem.

In an aspect, a negative microaneurysm confounder can be defined by alinear vessel portion. As an example, the negative microaneurysmconfounder can be model by:

${{linearVessel}\left( {x,{y;\alpha},\beta,\theta} \right)} = {{{profile}\left( {{{\frac{v}{\alpha}};\beta},{- 1}} \right)}.}$

In an aspect, a negative microaneurysm confounder can be defined by avessel branch: three linear segments, modeled by

${{{linearVessel}\left( {x,{y;\alpha},\beta,\theta} \right)} = {{profile}\left( {{{\frac{v}{\alpha}};\beta},{- 1}} \right)}},$can meet at the center of the model I, where the three intensitydistributions, I1, I2 and I3, are fused and can be empirically expressedas I=(I1+1)(I2+1)(I3+1)−1.

In an aspect, a negative microaneurysm confounder can be defined by avessel crossing: two vessels, modeled by

${{{linearVessel}\left( {x,{y;\alpha},\beta,\theta} \right)} = {{profile}\left( {{{\frac{v}{\alpha}};\beta},{- 1}} \right)}},$meet at the center of the model, where the two intensity distributions,I1 and I2, are fused. Since the two vessels overlap, the intensity isdarker at the center of the model I. As an example, the confounder canbe empirically expressed as:I=−√{square root over (I ₁ ² +I ₂ ²)}

In an aspect, positive lesion confounders for microaneurysm detectioncan be modeled by a translation of one or more of the followingfunctions:

$\begin{matrix}{{{typical}\left( {x,{y;\alpha_{1}},\alpha_{2},\beta,\theta} \right)} = {{profile}\left( {{r;\beta},\delta} \right)}} \\{{{linearVessel}\left( {x,{y;\alpha},\beta,\theta} \right)} = {{profile}\left( {{{\frac{v}{\alpha}};\beta},{- 1}} \right)}}\end{matrix}$

In an aspect, negative lesion confounders can be modeled for drusendetection. As an example, a pisciform intensity distribution can beused. As a further example, the pisciform intensity distribution can beused to model Stargardt's disease flecks, as illustrated below:

$\begin{matrix}{{{fleck}_{❘{u > 0}}\left( {x,{y;\alpha_{1}},\alpha_{2},\beta,\theta} \right)} = {{profile}\left( {{{r\; 1};\beta},\delta} \right)}} \\{{{fleck}_{❘{u \leq 0}}\left( {x,{y;\alpha_{1}},\alpha_{2},\beta,\theta} \right)} = {{profile}\left( {{{r\; 2};\beta},\delta} \right)}} \\{{r\; 1} = \sqrt{\left( \frac{u}{\alpha_{1}} \right)^{2} + \left( \frac{v}{\alpha_{2}} \right)^{2}}} \\{{{r\; 2} = \sqrt{\frac{u^{2} + v^{2}}{\alpha_{1}^{2}}}},}\end{matrix}$where fleck_(|u>0) (resp. fleck_(|u≤0)) is the tail (resp. head) of thegoldfish. As a further example, positive lesion confounders can bemodeled for drusen detection.

In an aspect, once the intensity distribution of the lesions and lesionconfounders are modeled, samples can be generated from each intensitydistribution model according to the exemplary method illustrated in FIG.3. Although lesion detection is used as an example, other objects ofinterest can be used in various domains.

In step 302, a scale for the generated samples can be selected based onan estimation of the typical size of the lesions and lesions confoundersin images. As an example, the generated image samples can be defined ascircular image patches of radius R times as wide as the typical lesionsize (e.g., R=3). As an example, a wide ratio R can be selected sincemost negative lesion confounders look similar to the lesions at thecenter of the patch. As a further example, lesions can be discriminatedfrom negative lesion confounders by the periphery of the patch. As afurther example, for microaneurysm detection, vessel crossings may looksimilar to a microaneurysm at the center, but, as opposed tomicroaneurysms, there will be linear structures, oriented toward thecenter of the patch, at the periphery.

In step 304, for each parameter π_(i) of a model (the standarddeviations α₁ and α₂ of the lesions, the angle between vessel segments,the distance between two neighboring lesions, etc.), a range [π_(i) min;π_(i) max] of variation can be estimated from the training set.

In step 306, image patches can be generated from each model by uniformlysampling the product spaceΠ_(i)[π_(i) ^(min); π_(i) ^(max)].Although many parameters might be normally distributed across actuallesions, a uniform sampling ensures that rare lesions are equally wellcharacterized. However, other samplings can be used.

In an aspect, a color model can be used to improve further the presentedgraylevel modeling of the lesions. As an example, gray levels in thegenerated samples are mapped to colors, using a look-up table (e.g.intensity 0—the background—can be mapped to pale orange and intensity−1—the center of vessels and microaneurysms—can be mapped to dark red,for example). If the scale range for the target lesion is veryimportant, then different sizes of patches can be generated.

In an aspect, an advanced physical model of the lesion transform, i.e.the effect the local pathology has on overlying and surrounding tissue,can be created, thus completely predicting the visual appearance of thelesions and lesion confounders. However, such a model would be highlycomplex, and more importantly, its design would require substantialdomain knowledge, as well as a substantial amount of experiments toperfect it for each new application. Modeling is even more challengingif color is involved: a direct sampling color model simply requiressampling in the entire color space, instead of complex modeling of theinteraction of color channel intensities. Hence, a data-driven or directsampling approach has many advantages.

In direct sampling, the target lesions (e.g., typical lesions andpositive lesion confounders) can be annotated on a training dataset. Asan example, the annotation comprises an indication of the center of thelesions, or segments the lesions. In an aspect, a candidate lesiondetector can be used to find a center of the lesion within the segmentedregion. As an example, a set of sample images can represent all imagesand all local image variation that the proposed filter framework mayencounter to optimize the framework.

Returning to FIG. 1, in an aspect, in order to increase therepresentativeness of the reference samples, one or more initial stepscan be performed, at 108. First, the variability due to the noisepotentially embedded in the reference samples can be minimized. Second,the generality of the reference samples can be increased by anormalization process.

In an aspect, the presence of low-frequency noise (slow backgroundintensity variation) and high-frequency noise (salt-and-pepper noise,compression artifacts, etc.) is typically a major problem forcharacterizing the appearance of lesions in retinal images. Accordingly,the noise-free samples S and/or noisy samples S+N can be projected intoa transformed space where the signal-to-noise ratio is high, such asT(S)≈T(S+

)≈T(S+

′), where T is the transform operator, and N and N′ are two realizationsof a random noise. As an example, image samples can be convolved withhorizontal and vertical Haar filters, at 108, dilated by a factor s tomatch the typical scale of the lesions (s=2 in our applications).

In an aspect, representative samples characterize the range ofvariations for the shape, the intensity and the color of the targetlesions and lesion confounders. One of these parameters, the intensityof the object can easily be normalized across samples, at 108.Normalizing this parameter reduces the dimensionality of the problem,without significantly affecting the quality of the image samplerepresentation. In particular, a higher generalization ability can beachieved with the same number of image samples (see FIG. 9). As aconsequence, less images need to be annotated. Normalizing the intensityof samples generated by a mathematical model can be trivial: theintensity at the center of the model simply needs to be set to one.Normalizing the intensity of samples directly extracted from images, onthe other hand, is not: for a reliable estimation of the lesionintensity, the intensity distribution in the target sample is matched tothe intensity distribution in the average sample using the least meansquared algorithm, in the transformed space of high signal-to-noiseratio. Finally, the intensity of each pixel in the transformed samplecan be divided by its estimated intensity.

In an aspect, the reference samples for lesions and lesion confounders,can be projected in the transformed space of high signal-to-noise ratioand subsequently normalized. As an example, the reference samples can bemaximally representative of all lesions and lesion confounders, providedthat these are also projected into the transformed space and normalized.In the absence of time constraints, classifying a new sample can bereduced to finding the most similar reference samples, in a least squaresense. In other words, the normalized transformed space described abovecan be used as feature space for classification with a k-NN classifier.Because it is desirable to achieve close to instantaneous detection, thedimensionality of the feature space can be configured to be as low aspossible. As an example, the reference samples can be relied upon tofind the filters generating the optimal feature space for classifyingnew samples: a reduced feature space preserving information in the leastsquare sense.

In an aspect, for any desired feature space dimension, the optimalprojection for given samples (the transformed and normalized referencesamples in our case), in terms of information preservation in leastsquare sense, can be provided by the Principal Component Analysis (PCA)of these samples. As an example, PCA is an orthogonal lineartransformation that transforms the set of samples into a new coordinatesystem such that the greatest variance, by any projection of the data,comes to lie on the first coordinate, the second greatest variance onthe second coordinate, etc.

In an aspect, each sample can be considered as a single observation,through applying PCA on the set of samples, principal components can beobtained which can be the same size as the samples. These principalcomponents can be the optimal filters because the linear combination ofthese filters represents the entire set of samples, and the coefficientsof the linear combination can be obtained by the convolution of originalsamples with corresponding filters.

The optimal filters thus found can span the initial, high dimensionalfeature space into which any image pixel can be projected. In an aspect,bifurcations in a training set can be projected into the feature spacein this way to form the positive reference points. Non-bifurcationpoints can be randomly sampled in the training set and projected intothe feature space in the same way to form the negative reference points.Both of bifurcation and non-bifurcation points together can form thereference points in the feature space. For computational efficiency,sampled pixels from the training set can be projected into the initialfeature space by convolving them with the filters for later usage.

In an aspect, given a set of optimal filters 110, previously unseenimages and image regions can be classified within this optimal featurespace. A so-called risk of presence of the target lesion can derived foreach of the image samples, at 112. This risk can subsequently be used tosegment the lesions in the image, or, to derive an automatedprobabilistic diagnosis for the entire image, at 114.

In an aspect, images can be processed based upon the method illustratedin FIG. 4. However, other sequences, steps, and processes, can be usedand applied in various domains.

In step 402, a preprocessing step can be used to identify and normalizecandidate target lesions in an image. In step 404, for each pixel pi;jselected in the preprocessing step 402, the normalized neighborhood ofthe pixel—the sample—is input to the classifier. As an example, the riskof presence of the target lesion is computed by the classifier for pi;jand a lesion risk map is thus obtained, at 406. In step 408, if thelesions need to be segmented, the risk map defined in step 406 isthresholded, and the connected foreground pixels are identified usingmorphological labeling. Each connected component is regarded as anautomatically detected lesion, and this lesion is assigned a risk valuedefined as the maximal pixelwise risk of presence within the connectedcomponent. In step 410, if a probabilistic diagnosis for the entireimage is desired/required, the risks assigned to each automaticallydetected lesions are fused. Typically, if a single type of lesions isdetected, the probabilistic diagnosis for the image is simply defined asthe maximum risk of presence of a target lesion.

Returning to FIG. 1, in an aspect, a candidate extraction step 116 isrequired for both selecting reference negative lesion confounders in thedirect sampling and selecting the pixels that should be fed to theclassifier when processing unseen images.

As more clearly shown in FIG. 5, the first step to extract candidates inan input image is to transform the image, at 502. As an example, thetechniques discussed at step 108 can be used to transform the image.However, other transforms can be used. For each pixel pi;j in thetransformed image, the intensity of the potential lesion within theneighborhood Ni;j of the pixel pi;j can then be estimated, at 504. As anexample, the techniques discussed at step 108 can be used to estimateintensity of pixels of the image. However, other estimation processesand methods can be used.

In step 506, the intensity of the potential legion can be compared to athreshold or pre-defined range value. If the intensity of the potentiallesion is outside the normal range of intensities for the targetlesions—the range observed on the training set plus a margin—, pi;j isrejected as a potential lesion. As an example, it is preferable todiscard candidates with low intensity (the background): by chance,magnified background pixels might look similar to the target lesions;similarly, attenuated pigment clumps or attenuated small nevi might looklike lesions, so candidates with an unusually high intensity should bediscarded. If the lesion intensity is within that range, theneighborhood Ni;j is normalized (i.e. divided by the estimatedintensity), at 508, before being further classified.

As an example, when this candidate extraction step is used to selectreference negative lesion confounders in the direct sampling approach,the candidates pixels pi;j are selected using a manually tuned constantthreshold on the average pixel-to-pixel squared distance to the averagetransformed and normalized positive lesion sample (=0.3). Note that thiscandidate extraction step is related to the classifier described in.

Returning to FIG. 1, in an aspect, since the set of filters generatingthe optimal feature space, at 118, can be derived from a representativeset of lesions and lesion confounders, a straightforward approach toclassify new image samples within the optimal feature space can be tofind the most similar representative samples within this space. A lesionrisk can be computed (e.g., at 112) for a new sample from the distanceto these representative samples and their label. As an example, thedistance between two samples is simply defined as the Euclidian distancebetween their projections on the optimal set of filters. Approximatenearest neighbor search, based on k-d trees, can be applied to find themost similar reference samples.

Once the k most similar samples (r_(i))_(i=1..k) of label(l(r_(i)))_(i=1..k) have been found, the presence risk ρ(s) is definedfor the new sample s as follows:

${\rho(s)} = {\frac{1}{k}{\sum\limits_{i = 1}^{k}{{l\left( r_{i} \right)}{\exp\left( {- {{r_{i} - s}}} \right)}}}}$where l(r_(i))=1 if r_(i) is a target lesion (typical lesion or commonpositive lesion confounder) and l(r_(i))=−1 if r_(i) is a commonnegative lesion confounder. This risk ranges from −1 (high risk of beingnegative) to 1 (high risk of being positive). If the new sample isneither a typical lesion nor a lesion confounder, then the distance tothe most similar reference samples is likely to be high. As aconsequence, a risk close to zero should be measured. In particular, ifbackground pixels are not discarded by the candidate selection step, itis likely that a risk close to zero is measured.

In an aspect, if the set of reference samples is representative of thenegative (resp. positive) lesion confounders, then false positives(resp. false negatives) should only occur for rare negative (resp.positive) lesion confounders. Accordingly, system performance and systemcomplexity can exchanged by adjusting our definition of “common” and“rare” lesion confounders.

In an aspect, selecting the optimal filters and generating the referencepoints in the feature space, the filters and reference points can beused to classify whether a pixel in a fundus image is a bifurcation. Animage under detection can be transferred into the feature space by beingconvolved with the optimal filters so each pixel corresponds to a querypoint. As an example, the k-Nearest Neighbor algorithm (k-NN) can beused to classify the query points. As a further example, ApproximateNearest Neighbor (ANN) can be applied to classify query points. In anaspect, soft classification and the probability of a query point to be abifurcation can be calculated by p=n=k, in which k means the number ofneighbors considered and n is the number of positive points among the knearest neighbors. Thus, a probability map of the original imagereflecting the probability of every pixel to be a bifurcation can beobtained. In an aspect, Even though the dimension of feature space canbe the number of chosen optimal filters (i.e. PCs), it is still a highdimensional space for classification. Accordingly, feature selection canbe applied to select a few of most discriminative and efficient filters.

In an aspect, probability maps can be generated with different optimalfilters chosen by Sequential Forward Selection (SFS) for each iteration.As an example, the metric can evaluate the probability maps and give theAUC for the ROC curve for the whole set of probability maps. As afurther example, SFS first selects out one filter with the highest AUC,then adds a new filter from the remaining filters such that the twofilters have the highest AUC. In an aspect, SFS can add a new filterfrom the remaining filters to give the highest AUC until the stopcriteria of the feature selection is met. As an example, the featureselection stops when the number of selected filters reaches the maximalnumber of filters, or the AUC starts to decline.

In an aspect, a metric to give the AUC can be pixel-based. Since theprobability map is soft labeled, the map can be thresholded to obtain abinary image and evaluate it pixel by pixel. As an example, every pixelP is decided as true positive (TP), false positive (FP), true negative(TN) or false negative (FN) according to its distance to a bifurcation,which is shown by

$P = \left\{ {\begin{matrix}{TP} & {{{if}\mspace{14mu} P\mspace{14mu}{is}\mspace{14mu}{labeled}\mspace{14mu}{and}\mspace{14mu}{{P - B}}} < D} \\{FP} & {{{if}\mspace{14mu} P\mspace{14mu}{is}\mspace{14mu}{labeled}\mspace{14mu}{and}\mspace{14mu}{{P - B}}} > D} \\{TN} & {{{if}\mspace{14mu} P\mspace{14mu}{is}\mspace{14mu}{unlabeled}\mspace{14mu}{and}\mspace{14mu}{{P - B}}} > D} \\{FN} & {{{if}\mspace{14mu} P\mspace{14mu}{is}\mspace{14mu}{unlabeled}\mspace{14mu}{and}\mspace{14mu}{{P - B}}} < D}\end{matrix},} \right.$where |P−B| is the distance between pixel P and any bifurcation B, and Ddefines the region of bifurcations.

As an example, when data is highly skewed that the number of negativepoints (non-bifurcation pixels) is much larger than that of positivepoints (i.e. bifurcation pixels) in a fundus image, so the ROC isinsensitive to the change of specificity due to the large number of truenegative, i.e. unlabeled non-bifurcation pixels. For betterdiscriminability of ROC curve, all positive points can be weighed with afactor F larger than 1 by the following equation:

$F = {2 \times \sqrt{\frac{{total}\mspace{14mu}{{No}.\mspace{14mu}{of}}\mspace{14mu}{negative}}{{total}\mspace{14mu}{{No}.\mspace{14mu}{of}}\mspace{14mu}{positive}}}}$

In an aspect, introduction of factor F can enhance the influence ofpositive points so the ROC curve is more sensitive to changes ofsensitivity and specificity. Assuming there are n images used in featureselection, the metric giving the AUC for the probability maps isoutlined as follows:

-   -   a) find the threshold values v0, v1, v2, . . . vk for all        probability maps;    -   b) for threshold value i=v0, . . . vk        -   1. threshold all probability maps and obtain thresholded            images mi1, mi2, . . . min;        -   2. compute TP, FP, TN and FN for mi1, mi2, . . . min;        -   3. sum all TP, FP, TN and FN together for all thresholded            images; and/or        -   4. calculate the sensitivity and specificity for threshold            value I; and/or    -   c) compute the AUC according to the pairs of sensitivity and        specificity for all threshold values.

As described in further detail below, one skilled in the art willappreciate that this is a functional description and that the respectivefunctions can be performed by software, hardware, or a combination ofsoftware and hardware. A unit can be software, hardware, or acombination of software and hardware. The units can comprise theprocessing Software 606 as illustrated in FIG. 6 and described below. Inone exemplary aspect, the units can comprise a computer 601 asillustrated in FIG. 6 and described below.

FIG. 6 is a block diagram illustrating an exemplary operatingenvironment for performing the disclosed methods. This exemplaryoperating environment is only an example of an operating environment andis not intended to suggest any limitation as to the scope of use orfunctionality of operating environment architecture. Neither should theoperating environment be interpreted as having any dependency orrequirement relating to any one or combination of components illustratedin the exemplary operating environment.

The present methods and systems can be operational with numerous othergeneral purpose or special purpose computing system environments orconfigurations. Examples of well known computing systems, environments,and/or configurations that can be suitable for use with the systems andmethods comprise, but are not limited to, personal computers, servercomputers, laptop devices, and multiprocessor systems. Additionalexamples comprise set top boxes, programmable consumer electronics,network PCs, minicomputers, mainframe computers, distributed computingenvironments that comprise any of the above systems or devices, and thelike.

The processing of the disclosed methods and systems can be performed bysoftware components. The disclosed systems and methods can be describedin the general context of computer-executable instructions, such asprogram modules, being executed by one or more computers or otherdevices. Generally, program modules comprise computer code, routines,programs, objects, components, data structures, etc. that performparticular tasks or implement particular abstract data types. Thedisclosed methods can also be practiced in grid-based and distributedcomputing environments where tasks are performed by remote processingdevices that are linked through a communications network. In adistributed computing environment, program modules can be located inboth local and remote computer storage media including memory storagedevices.

Further, one skilled in the art will appreciate that the systems andmethods disclosed herein can be implemented via a general-purposecomputing device in the form of a computer 601. The components of thecomputer 601 can comprise, but are not limited to, one or moreprocessors or processing units 603, a system memory 612, and a systembus 613 that couples various system components including the processor603 to the system memory 612. In the case of multiple processing units603, the system can utilize parallel computing.

The system bus 613 represents one or more of several possible types ofbus structures, including a memory bus or memory controller, aperipheral bus, an accelerated graphics port, and a processor or localbus using any of a variety of bus architectures. By way of example, sucharchitectures can comprise an Industry Standard Architecture (ISA) bus,a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, aVideo Electronics Standards Association (VESA) local bus, an AcceleratedGraphics Port (AGP) bus, and a Peripheral Component Interconnects (PCI),a PCI-Express bus, a Personal Computer Memory Card Industry Association(PCMCIA), Universal Serial Bus (USB) and the like. The bus 613, and allbuses specified in this description can also be implemented over a wiredor wireless network connection and each of the subsystems, including theprocessor 603, a mass storage device 604, an operating system 605,processing software 606, processing data 607, a network adapter 608,system memory 612, an Input/Output Interface 610, a display adapter 609,a display device 611, and a human machine interface 602, can becontained within one or more remote computing devices 614 a,b,c atphysically separate locations, connected through buses of this form, ineffect implementing a fully distributed system.

The computer 601 typically comprises a variety of computer readablemedia. Exemplary readable media can be any available media that isaccessible by the computer 601 and comprises, for example and not meantto be limiting, both volatile and non-volatile media, removable andnon-removable media. The system memory 612 comprises computer readablemedia in the form of volatile memory, such as random access memory(RAM), and/or non-volatile memory, such as read only memory (ROM). Thesystem memory 612 typically contains data such as processing data 607and/or program modules such as operating system 605 and processingsoftware 606 that are immediately accessible to and/or are presentlyoperated on by the processing unit 603.

In another aspect, the computer 601 can also comprise otherremovable/non-removable, volatile/non-volatile computer storage media.By way of example, FIG. 6 illustrates a mass storage device 604 whichcan provide non-volatile storage of computer code, computer readableinstructions, data structures, program modules, and other data for thecomputer 601. For example and not meant to be limiting, a mass storagedevice 604 can be a hard disk, a removable magnetic disk, a removableoptical disk, magnetic cassettes or other magnetic storage devices,flash memory cards, CD-ROM, digital versatile disks (DVD) or otheroptical storage, random access memories (RAM), read only memories (ROM),electrically erasable programmable read-only memory (EEPROM), and thelike.

Optionally, any number of program modules can be stored on the massstorage device 604, including by way of example, an operating system 605and processing software 606. Each of the operating system 605 andprocessing software 606 (or some combination thereof) can compriseelements of the programming and the processing software 606. Processingdata 607 can also be stored on the mass storage device 604. Processingdata 607 can be stored in any of one or more databases known in the art.Examples of such databases comprise, DB2®, Microsoft® Access, Microsoft®SQL Server, Oracle®, mySQL, PostgreSQL, and the like. The databases canbe centralized or distributed across multiple systems.

In another aspect, the user can enter commands and information into thecomputer 601 via an input device (not shown). Examples of such inputdevices comprise, but are not limited to, a keyboard, pointing device(e.g., a “mouse”), a microphone, a joystick, a scanner, tactile inputdevices such as gloves, and other body coverings, and the like These andother input devices can be connected to the processing unit 603 via ahuman machine interface 602 that is coupled to the system bus 613, butcan be connected by other interface and bus structures, such as aparallel port, game port, an IEEE 1394 Port (also known as a Firewireport), a serial port, or a universal serial bus (USB).

In yet another aspect, a display device 611 can also be connected to thesystem bus 613 via an interface, such as a display adapter 609. It iscontemplated that the computer 601 can have more than one displayadapter 609 and the computer 601 can have more than one display device611. For example, a display device can be a monitor, an LCD (LiquidCrystal Display), or a projector. In addition to the display device 611,other output peripheral devices can comprise components such as speakers(not shown) and a printer (not shown) which can be connected to thecomputer 601 via Input/Output Interface 610. Any step and/or result ofthe methods can be output in any form to an output device. Such outputcan be any form of visual representation, including, but not limited to,textual, graphical, animation, audio, tactile, and the like.

The computer 601 can operate in a networked environment using logicalconnections to one or more remote computing devices 614 a,b,c. By way ofexample, a remote computing device can be a personal computer, portablecomputer, a server, a router, a network computer, a peer device or othercommon network node, and so on. Logical connections between the computer601 and a remote computing device 614 a,b,c can be made via a local areanetwork (LAN) and a general wide area network (WAN). Such networkconnections can be through a network adapter 608. A network adapter 608can be implemented in both wired and wireless environments. Suchnetworking environments are conventional and commonplace in offices,enterprise-wide computer networks, intranets, and the Internet 615.

For purposes of illustration, application programs and other executableprogram components such as the operating system 605 are illustratedherein as discrete blocks, although it is recognized that such programsand components reside at various times in different storage componentsof the computing device 601, and are executed by the data processor(s)of the computer. An implementation of processing software 606 can bestored on or transmitted across some form of computer readable media.Any of the disclosed methods can be performed by computer readableinstructions embodied on computer readable media. Computer readablemedia can be any available media that can be accessed by a computer. Byway of example and not meant to be limiting, computer readable media cancomprise “computer storage media” and “communications media.” “Computerstorage media” comprise volatile and non-volatile, removable andnon-removable media implemented in any methods or technology for storageof information such as computer readable instructions, data structures,program modules, or other data. Exemplary computer storage mediacomprises, but is not limited to, RAM, ROM, EEPROM, flash memory orother memory technology, CD-ROM, digital versatile disks (DVD) or otheroptical storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices, or any other medium which canbe used to store the desired information and which can be accessed by acomputer.

The methods and systems can employ Artificial Intelligence techniquessuch as machine learning and iterative learning. Examples of suchtechniques include, but are not limited to, expert systems, case basedreasoning, Bayesian networks, behavior based AI, neural networks, fuzzysystems, evolutionary computation (e.g. genetic algorithms), swarmintelligence (e.g. ant algorithms), and hybrid intelligent systems (e.g.Expert inference rules generated through a neural network or productionrules from statistical learning).

EXAMPLES

The following examples are put forth so as to provide those of ordinaryskill in the art with a complete disclosure and description of how thecompounds, compositions, articles, devices and/or methods claimed hereinare made and evaluated, and are intended to be purely exemplary and arenot intended to limit the scope of the methods and systems. Efforts havebeen made to ensure accuracy with respect to numbers (e.g., amounts,numbers of samples, etc.), but some errors and deviations should beaccounted for.

As an example, the proposed system was tested on microaneurysm detectionfor DR screening and drusen detection for AMD detection. If notmentioned otherwise, the computations were performed in the greenchannel of fundus images.

A. Microaneurysm Detection

The performance of the proposed framework at detecting microaneurysmswas first evaluated at a lesion level, on a set of 100 deidentifiedimages manually segmented by a clinician expert (MDA). Twenty-six ofthese images were taken from a DR screening program in the Netherlands;these images were originally captured using a Canon CR5 nonmydriatic3CCD camera at 45 degree field of view; and 74 were obtained Dr.Abramoff's retinal referral clinic; these images were captured using aTopcon TRC-50 at 45 degree field of view.

The size of the images is 768×576 pixels and the field of view isapproximately 540 pixels in diameter. 55/100 of the images containedlesions. In these images, a total of 858 red lesions were identified.

Half of the images in this dataset were used for training and the otherhalf for testing. In order to validate the optimal fitting of thefilters to the reference samples, the optimal filtering framework(including confounders) was also compared to an approach were analyticalfilters are selected to maximize the classification accuracy, bycross-validation, on the set of reference samples (‘training set’). Theanalytical filters used in this experiment were typical: Gaussianderivatives and Gabor wavelets in color opponency space. All otheraspects of the method (convolution and normalization of the data, k-NNclassifier with the same risk) remained unchanged. The radius of thefilters was set to 9 pixels for this application. The optimal filtersare reported in FIGS. 7A-7B, wherein The upper row of each filter isconvolved with vertical Haar filters dilated by a factor of 2 and thelower row of each filter is convolved with horizontal Haar filtersdilated by a factor of 2. As an example, FIGS. 7A-7B illustrate thatsimilar sets of filters can be obtained regardless of the method that isused to collect the set of reference samples.

In an aspect, FIG. 8 illustrates an exemplary microaneurysm detection(e.g., detection image 804) by the proposed mathematical model basedmethod in the macular region of a JPEG-compressed image 802 used in theEyeCheck screening program.

The free-response receiver-operating characteristic (FROC) curves of thedifferent methods on the testing set are reported in FIG. 9, wherein the“Expertdriven approach—no confounders” shows the performance obtainedwith a simplified version of the method where the confounder sampleswere not in the training set; there were only 2 reference samples: asmall and a large typical lesion. “Data-driven approach—any negativesamples” shows the performance obtained if the negative referencesamples are selected at random outside of the manually segmentedregions, instead of being selected among negative lesion confounders(i.e. among false positives of candidate selection). “Data-drivenapproach—no normalization” shows the performance obtained if thenormalization step is omitted both for training and testing.

The time complexity assessment of the method is presented in FIGS.10A-10D, wherein the computation times were obtained on a standard PCrunning at 2.26 GHz. Each vertex in FIG. 10A from left to right,corresponds to one curve in FIG. 10B from “299 samples” to “61373samples.” Likewise, each vertex in FIG. 10C from left to right,corresponds to one curve in FIG. 10D from “5 PCSs” to “200 PCs.”

For convenience, the time complexity of the mathematical model basedapproach is studied (it is easier to increase the number of generatedsamples); similar computations times are obtained with the directsampling approach.

Once the proposed method was calibrated at a lesion level, theperformance of the best performing version of the proposed algorithm(the mathematical-model approach) was compared at an examination levelto a re-implementation of Niemeijer's method. This experiment wascarried out on a set of images from the same screening program. The sizeof images and that of the field of view are similar to the previous set.2,739 consecutive examinations from diabetic patients on their firstvisit were selected from a set of first visits in 2008. Each examinationconsists of two images per eye (a macula-centered and an optic disccentered image).

Sixty-seven of them were diagnosed with DR by a single retinalspecialist, and the maximal microaneurysm presence risk measured withinthis set of four images was used as DR risk. The receiver-operatingcharacteristic (ROC) of these two methods are reported in FIG. 11,wherein ROC of microaneurysm detection at an image level on a set of2,739 patient images, including 67 with referable DR.

B. Drusen Detection

In the second application of the proposed framework, drusen aredifferentiated from Stargardt's disease flecks. Thus, the lesions andlesion confounders appear in different image sets, and the problem is todifferentiate the two types of lesions as to whether they are typicalfor AMD (drusen) or Stargardt's disease (flecks). A set of 15 images ofpatients with AMD, all containing drusen, as confirmed by a clinicianexpert (MDA), and 15 images containing flecks as confirmed to haveStargardt's disease by positive genetic testing for the ABCA4 haplotypewere selected. Images were acquired with a Zeiss FF3 30-degree cameraand recorded on Kodachrome film (ASA 100). Film-based images werescanned at 3456×2304 pixels, 12 bit depth, using a 3 color channel CCDscanner.

A set of 300 drusen and 300 flecks were manually annotated in theseimages and used as reference samples for the sampling approach. Aleave-one-image-out cross-validation was performed to assess thesampling approach. To validate the mathematical model approach, the 600samples were used as a testing set. The radius of the filters was set to13 pixels for this application. The optimal filters are reported inFIGS. 7C-7D. The ROC-s are reported in FIG. 12.

Accordingly, a novel, optimal filter framework for detecting targetlesions in retinal images has been provided. The framework can comprisea design of a set of filters optimally representing the typical targetlesions, as well as the negative lesion confounders, i.e. lesions thatare similar looking but not the target lesions, and the positive lesionconfounders, i.e. lesions that are easily missed because they havespecific properties that are substantially different from the standardcase. In order to design the optimal filter set, image samplesrepresentative of the three types of lesions are collected and thefilters are defined as the principal components of these samples.

Special attention is given in the proposed approach to ensure therepresentativeness of the reference image samples. Two completelydifferent approaches for obtaining these maximally representativesamples are proposed and compared: an expert and a data-driven approach.The optimal filter set is used as a reference to classify new imagesamples. Because the feature space spanned by these filters is compact,the classification procedure, based on an approximate nearest neighborsearch, is fast. This property is particularly important in medicalapplications such as the instantaneous assessment of retinal diseases.

The optimal filter framework has been tested on the detection of twoimportant lesions in eye disease: microaneurysms and drusen. Our resultsshow that the detection performance achieved by the new optimal filterframework is comparable to that of previous algorithms presented,however the new method is much faster: each retinal image is processedin less than a second, as opposed to several minutes with the previousapproach. This implies that instantaneous assessment of retinal imagesfor DR is achievable, while maintaining the performance achieved bypreviously published algorithms. The results show that the optimalfilter framework also has good performance in the differentiation ofdrusen from Stargardt's disease flecks, illustrating its generality.

The optimal filter sets obtained varied little with the method throughwhich they were obtained. In other words, it mattered little whether therepresentative samples (from which the optimal filters are derived) areobtained from a mathematical model designed by a non-clinician domainexpert, i.e. expert driven, or by directly sampling a representative setof images at locations annotated by one or more expert clinicians, i.e.data-driven, following the sampling procedure described in this paper.Thus, an optimal filter framework, with the set of domain optimalfilters obtained directly from image samples, and only requiring expertclinician input for annotating, performs close (for DR detection) orequivalent (for drusen versus flecks differentiation) to a system inwhich the filters require the design of a model by a non-clinicianexpert. The data-driven approach is expected to generalize well to otherretinal image analysis problems, and possibly, to other medical imageanalysis problems, because only limited clinician expert annotationeffort is required, and no further domain knowledge is required on thedesigner of the framework. Its generality and lack of bias isillustrated also by the fact that incorporating color into therepresentative samples results in equal or superior performance comparedto samples from a specific color channel, (the green channel, which isknown by domain experts to be more discriminant). If the expert modelerhad been biased and assumed that color was irrelevant, and therefore didnot incorporate color into their mathematical model, that might haveunnecessarily limited the performance. For the data-driven form of theoptimal filter framework there is no need to exclude potentiallyirrelevant color channels a priori. The expert-driven approach wassuperior to the data-driven approach when the designer of themathematical models had sound non-clinician expert knowledge(microaneurysm detection). It shows that, if expert knowledge isavailable, it is possible to push performance further throughmathematical modeling. Moreover, should the method be applied to a newdataset, the value of a few parameters could be changed, while in thedata-driven approach additional annotation might have been required.

Finally, the results indicate that using a set of filters obtained fromfactorial coding of the data—in our case, PCA—may be superior, and atleast equal, to the set of filters obtained through feature selection ona Gaussian derivative or Gabor wavelet or filterbank. As an example,from all of the potential filter sets in the filter, or feature, space,a small set Gabor or Gaussian derivative filters may be optimal, butthat for specific object detection problems, a locally optimal, betterperforming filter set exists, and can be found in this space through theframework described herein.

C. Bifurcation Detection

In an exemplary experiment, a total of 80 de-identified retinal colorimages from 80 subjects from a diabetic retinopathy screening program inthe Netherlands were created by registration of 2 separate images perpatients. The size of images was approximately 800×650 pixels in JPEGformat. The number of images in a first training set was 50, and thenumber of images in a second training set was 10. The number of testimages was 20. To build and evaluate a vision system or vision machine,an expert annotated a set of centers of bifurcations in the trainingsets and test images and a retinal specialist reviewed these annotationsfor correctness. After annotation, all bifurcation were samples at ascale 6 of 25×25 pixels. In the training stage, 10 bifurcations wererandomly sampled per image so 500 samples are created, and rotating thesamples resulted in 4000 samples. PCA was applied separately on thesamples in red, blue and green channels. In an aspect, images in redchannels contain the most amount of information and least amount ofnoise because its cumulative variance(CV) curve rises most rapidly.

As an example, the information of bifurcation samples are concentratedin the first few number of PCs. Thus the first 30 PCs of red and greenchannels can be selected respectively for feature selection. For thenegative points, 2000 non-bifurcations are selected from each trainingimage so there will be 100000 negative points in the feature space. Thisnumber is chosen in the consideration that the ratio of true bifurcationand true non-bifurcation is very large, accordingly, a large amount ofnegative points can be selected to simulate this situation.

In the feature selection step, the maximal number of optimal filters isset to be 10. The distance D that defines the region of true bifurcationis the square root of 41. In this case the 10 most discriminativefilters are 6th PC in red, and 2nd, 3rd, 4th, 5th, 6th, 7th, 18th and19th PCs in green channels. It is worth noticing that the selectedfilters depends on the distribution of the reference points and sincethe negative points are randomly sampled for different reference pointsfeature selection might give different results.

Using these 10 filters, a 10-dimensional feature space was established,within which there are 4000 positive points and 100000 negative points.A k-NN algorithm was used to classify 20 test images with K=33. Themetric used to evaluate the results of the 20 test images was the sameas in feature selection. The overall AUC was 0.883.

The results show that a vision machine based retinal bifurcationdetection method can be developed based solely on a limited number ofexpert annotated retinal images with exemplar branching points. Theresults also show that detection performance of the system, compared toa human expert, on a different dataset, as measured by AUC, issatisfactory.

In an aspect, using a vision machine approach as described herein, anautomated retinal vessel bifurcation detection system, based on verylimited expert contribution. The vision machine, systems and methodsdescribed herein can be more widely used to develop simple object searchsystems for similar applications.

In an aspect, similar approaches (e.g., optimal filter or visionmachine) can be used to create detection systems for various objects ofinterest. As an example, the systems and methods described herein can beused to detect apples in images of orchards, helicopter landing pads inhigh altitude reconnaissance images, and cell nuclei in phase contrastimages of mitotic fibroblasts. Other images and objects of interest canbe processed. As a further example, the methods described herein can behighly general but can be trained to be highly specific in a specificdomain. Any domain can be used and the systems and methods can betrained for any particular object of interest.

As described herein, a method is disclosed for identifying an object ofinterest in a plurality of digital images (e.g., ImageORG), where eachof these images comprises a plurality of wavelength bands, and eachImageORG comprises a plurality of raw images that were combined into asingle ImageORG. As an example, the method can comprise obtaining firstsamples of the intensity distribution of the object of interest in oneor more wavelength bands, obtaining second samples of the intensitydistribution of confounder objects, at a frequency high enough to affecta performance metric, transforming the first and second samples into anappropriate space, thereby creating Imagetrans, performing dimensionreduction on the transformed first and second samples altogether,whereby the dimension reduction of the transformed first and secondsamples creates an object detector, transforming ImageORG into theappropriate space to obtain Imagetrans, projecting Imagetrans into thespace of reduced dimension to obtain Imagereduc, classifying each pixelp in Imagereduc with an appropriate neighborhood Np, based on acomparison with the samples obtained, and automatically identifying anobject of interest from the abnormal pixels.

In summary, a novel, general, optimal filter framework for the detectionof lesions in retinal images, derived through factor analysis of thelesions and their confounders, has been presented in this paper and wassuccessfully applied to two important retinal image analysis problems.Potentially, this optimal filter framework can be applied to otherlesion detection problems in retinal images, where rapid systemdevelopment and instantaneous performance are required.

While the methods and systems have been described in connection withpreferred embodiments and specific examples, it is not intended that thescope be limited to the particular embodiments set forth, as theembodiments herein are intended in all respects to be illustrativerather than restrictive.

Unless otherwise expressly stated, it is in no way intended that anymethod set forth herein be construed as requiring that its steps beperformed in a specific order. Accordingly, where a method claim doesnot actually recite an order to be followed by its steps or it is nototherwise specifically stated in the claims or descriptions that thesteps are to be limited to a specific order, it is no way intended thatan order be inferred, in any respect. This holds for any possiblenon-express basis for interpretation, including: matters of logic withrespect to arrangement of steps or operational flow; plain meaningderived from grammatical organization or punctuation; the number or typeof embodiments described in the specification.

Throughout this application, various publications are referenced. Thedisclosures of these publications in their entireties are herebyincorporated by reference into this application in order to more fullydescribe the state of the art to which the methods and systems pertain.

It will be apparent to those skilled in the art that variousmodifications and variations can be made without departing from thescope or spirit. Other embodiments will be apparent to those skilled inthe art from consideration of the specification and practice disclosedherein. It is intended that the specification and examples be consideredas exemplary only, with a true scope and spirit being indicated by thefollowing claims.

The invention claimed is:
 1. A method for diagnosing a disease conditionin a patient, the method comprising: receiving an input image of aportion of a patient's body; accessing a supervised procedure configuredto output a probability that a digital image contains an object ofinterest at each of one or more locations within the digital image,where the supervised procedure is trained based on a set of trainingimages that are manually annotated to indicate whether the trainingimages contain a feature of interest at one or more locations therein;applying the input image to the supervised procedure to output, for eachof a set of reference locations in the input image, a probability thatthe input image contains an object of interest at the referencelocation, where one or more of the objects of interest are indicative ofthe disease condition; determining a feature space from theprobabilities outputted from the supervised procedure, the feature spacedescribing the set of reference locations in the input image and thecorresponding probabilities that each of the set of reference locationscontains an object of interest; passing the feature space to anautomated diagnostic model trained to provide a probabilistic diagnosisof the disease condition for the input image; and outputting theprobabilistic diagnosis of the disease condition from the automateddiagnostic model.
 2. The method of claim 1, wherein applying the inputimage to the supervised procedure comprises: obtaining one or moresamples for each of a set of different regions of the input image; andapplying each of the obtained samples of the input image to thesupervised procedure to output a probability that the sample containsone or more objects of interest, where the object of interest isindicative of the disease condition.
 3. The method of claim 1, whereinthe supervised procedure comprises a plurality of trained filters, eachtrained filter configured to identify a probability that the samplecontains an object of interest.
 4. The method of claim 1, wherein thereceived input image is of a portion of the patient's eye, and whereinthe disease condition comprises a disorder manifesting in the eye. 5.The method of claim 1, wherein the received input image is of a portionof the patient's eye, and wherein the disease condition comprises adisorder manifesting in the retina.
 6. The method of claim 1, whereinthe one or more objects of interest comprise an object that has asimilar appearance to another object that indicates a disease conditionin a patient but that does not itself indicate a disease condition in apatient.
 7. The method of claim 1, wherein the one or more objects ofinterest comprise an object selected from a group consisting of: amicroaneurysm, a dot hemorrhage, a flame-shaped hemorrhage, asub-intimal hemorrhage, a sub-retinal hemorrhage, a pre-retinalhemorrhage, a micro-infarction, a cotton-wool spot, a yellow exudate,and a drusen.
 8. The method of claim 1, wherein each of the set oftraining images is manually annotated.
 9. The method of claim 1, whereineach of the set of training images is annotated to indicate one or morereference points that correspond to a center of a feature of interest inthe training image.
 10. The method of claim 1, wherein the feature spacedescribes a reference location in the input image and correspondingprobability that the reference location contains an object of interestfor each of a plurality of objects of interest of different types.
 11. Acomputer program product for diagnosing a disease condition in apatient, the computer program product comprising a non-transitorycomputer-readable storage medium containing computer program code for:receiving an input image of a portion of a patient's body; accessing asupervised procedure configured to output a probability that a digitalimage contains an object of interest at each of one or more locationswithin the digital image, where the supervised procedure is trainedbased on a set of training images that are annotated to indicate whetherthe training images contain a feature of interest at one or morelocations therein; applying the input image to the supervised procedureto output, for each of a set of reference locations in the input image,a probability that the input image contains an object of interest at thereference location, where one or more of the objects of interest areindicative of the disease condition; determining a feature space fromthe probabilities outputted from the supervised procedure, the featurespace describing the set of reference locations in the input image andthe corresponding probabilities that each of the set of referencelocations contains an object of interest; passing the feature space toan automated diagnostic model trained to provide a probabilisticdiagnosis of the disease condition for the input image; and outputtingthe probabilistic diagnosis of the disease condition from the automateddiagnostic model.
 12. The computer program product of claim 11, whereinapplying the input image to the supervised procedure comprises:obtaining one or more samples for each of a set of different regions ofthe input image; and applying each of the obtained samples of the inputimage to the supervised procedure to output a probability that thesample contains one or more objects of interest, where the object ofinterest is indicative of the disease condition.
 13. The computerprogram product of claim 11, wherein the supervised procedure comprisesa plurality of trained filters, each trained filter configured toidentify a probability that the sample contains an object of interest.14. The computer program product of claim 11, wherein the received inputimage is of a portion of the patient's eye, and wherein the diseasecondition comprises a disorder manifesting in the eye.
 15. The computerprogram product of claim 11, wherein the received input image is of aportion of the patient's retina, and wherein the disease conditioncomprises a disorder manifesting in the retina.
 16. The computer programproduct of claim 11, wherein the one or more objects of interestcomprise an object that has a similar appearance to another object thatindicates a disease condition in a patient but that does not itselfindicate a disease condition in a patient.
 17. The computer programproduct of claim 11, wherein the one or more objects of interestcomprise an object selected from a group consisting of: a microaneurysm,a dot hemorrhage, a flame-shaped hemorrhage, a sub-intimal hemorrhage, asub-retinal hemorrhage, a pre-retinal hemorrhage, a micro-infarction, acotton-wool spot, a yellow exudate, and a drusen.
 18. The computerprogram product of claim 11, wherein each of the set of training imagesis manually annotated.
 19. The computer program product of claim 11,wherein each of the set of training images is annotated to indicate oneor more reference points that correspond to a center of a feature ofinterest in the training image.
 20. The computer program product ofclaim 11, wherein the feature space describes a reference location inthe input image and corresponding probability that the referencelocation contains an object of interest for each of a plurality ofobjects of interest of different types.